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ABSTRACT 

This paper proposes to generate and to use barcodes to an¬ 
notate medical images and/or their regions of interest such 
as organs, tumors and tissue types. A multitude of efficient 
feature-based image retrieval methods already exist that can 
assign a query image to a certain image class. Visual anno¬ 
tations may help to increase the retrieval accuracy if com¬ 
bined with existing feature-based classification paradigms. 
Whereas with annotations we usually mean textual descrip¬ 
tions, in this paper barcode annotations are proposed. In 
particular, Radon barcodes (RBC) are introduced. As well, 
local binary patterns (LBP) and local Radon binary patterns 
(LRBP) are implemented as barcodes. The IRMA x-ray 
dataset with 12,677 training images and 1,733 test images is 
used to verify how barcodes could facilitate image retrieval. 

Index Terms — Medical image retrieval, annotation, bar¬ 
codes, Radon transform, binary codes, local binary pattern. 

1. IDEA AND MOTIVATION 

The idea proposed in this paper is to generate short barcodes, 
embed them in the medical images (i.e. in DICOM files) 
and use them, along with feature-based methods, for fast and 
accurate image search. Retrieving (similar) medical images 
when a query image is given could assist clinicians for more 
accurate diagnosis by comparing with similar (retrieved) 
cases. As well, image retrieval can contribute to research- 
oriented tasks in biomedical imaging in general (e.g. in 
histopathology). 

Why not features? This proposal does not seek to replace 
feature-based classification approach to content-based image 
retrieval (CBIR). Paradigms such as “bag of words” and “bag 
of features” along with powerful classifiers such as SVM and 
KNN have undoubtedly moved the research forward in im¬ 
age search. Barcode annotations should solely assist in this 
process, particularly for medical images, as additional source 
of information. Even though the retrieval capabilities of bar¬ 
codes will be examined in this paper, they are not meant to be 
the main retrieval mechanism but auxiliary to existing ones. 

The author would like to thank NSERC (Natural Sciences and Engineer¬ 
ing Research Council of Canada) for funding this project (Discovery Grant). 


Why barcodes? As supplementary information, barcodes 
could enhance the results of existing “bag-of-features” meth¬ 
ods, which are generally designed to capture the global ap¬ 
pearance of the scene without much attention to the local de¬ 
tails of scene objects (e.g. shape of a tumor in an MR scan). 
Specially local, ROI-based barcodes may be more expressive 
in capturing spatial information. Whether ID or 2D barcodes 
are used may deliver the same results even though ID bar¬ 
codes are expected to be shorter and hence faster in execution. 

2. LITERATURE REVIEW 

Since Radon barcodes will be introduced in this paper, and be¬ 
cause local binary patterns are implemented for sake of com¬ 
parison, in following the relevant literature will be briefly re¬ 
viewed. Due to space limitations we cannot review the vast 
literature on CBIR as adequately as we generally do. 

Literature on Radon transform - Depicting a three di¬ 
mensional object is the main motivation for Radon transform. 
There are many applications of Radon transform reported 
in literature. Zhao et al. □ use feature detection in Radon 
transformed images to calculate the ocean wavelength and 
wave direction in SAR images. Hoang and Tabbone 0 
employed Radon transform in conjunction with Fourier and 
Mellin transform for extraction of invariant features. Nac- 
ereddine et al. E) also used Radon transform to propose a 
new descriptor called Phi-signature for retrieval of simple 
binary shapes. Jadhav and Holambe 10 use Radon trans¬ 
form along with discrete wavelet transform to extract features 
for face recognition. Chen and Chen a introduced Radon 
composite features (RCFs) that transform binary shapes into 
ID representations to calculate features from. Tabbone et 
al. 0 propose a histogram of the Radon transform (HRT), 
which is invariant to geometrical transformations. They use 
HRT for shape retrieval. Dara et al. (TJ generalized Radon 
transform to radial and spherical integration to search for 3D 
models of diverse shapes. Trace transform is also a gener¬ 
alization of Radon transform (313 for invariant features via 
tracing lines applied on shapes with complex texture on a 
uniform background for change detection. Although binary 
images/thumbnails have been used to facilitate image search 
ESHH2CC2, it seems that no attempt has been made to bi- 
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narize the Radon projections and use them directly for CBIR 
tasks as it will be proposed in this paper. 

Literature on LBPs and CBIR - Local binary patterns 
(LBPs) were introduced by Ojala et al. lfl4ll . Among others, 
LBPs have been used for face recognition OilCDD. The liter- 
ature on CBIR and medical CBIR is vast. Ghosh et al. El 
review online systems for content-based medical image re¬ 
trieval such as GoldMiner, BioText, FigureSearch, Yottalook, 
Yale Image Finder, IRMA and iMedline. Multiple surveys are 
available that review recent literature lfl~8l IT9l [20l. 
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3. RADON BARCODE ANNOTATIONS 



Examining a function f(x,y ), one can project f(x,y) along 
a number of projection angles. The projection is basically 
the sum (integral) of /(x, y) values along lines constituted by 
each angle 6. The projection creates a new image R(p , 6) with 
p = x cos 0 + y sin 0. Hence, using the Dirac delta function 
S(-) the Radon transform can be written as 

+oo +oo 

R(p,e) = // f(x,y)6(p — xcosO — ysin 0)dxdy. (1) 
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Fig. 1. Radon Barcode (RBC) - The image is Radon- 
transformed. All projections (P1,P2,P3,P4) are thresholded 
to generate code fragments C1,C2,C3,C4. The concatenation 
of all code fragments delivers the barcode RBC. 


— oo —oo 


If we threshold all projections (lines) for individual angles 
based on a “local” threshold for that angle, then we can as¬ 
semble a barcode of all thresholded projections as depicted 
in Figure [I] A simple way for thresholding the projections 
is to calculate a typical value via median operator applied on 
all non-zero values of each projection. Algorithm [T] describes 
how Radon barcodes (RBC) are generated^] In order to re¬ 
ceive same-length barcodes Normalize(I) resizes all images 
into Rn x Cn images (i.e. Rn = Cn = 2 n , n G N + ). 


Algorithm 1 Radon Barcode (RBC) Generation 
1: Initialize Radon Barcode r <— 0 
2: Initialize angle 6 0 and Rn = Cn 32 

3: Normalize the input image I = Normalize^, Rn,Cn) 
4: Set the number of projection angles, e.g. n p <— 8 
5: while 0 < 180 do 
6: Get all projections p for 6 

7: Find typical value T typica i <- median; (p; ) | p , /o 

8: Binarize projections: b P > ^typical 

9: Append the new row r <— append(r, b) 

10 : 0 <- 0 + ±30 

Up 

11: end while 
12: Return r 


4. LBP BARCODE ANNOTATIONS 

Local binary patterns (LBPs) have been extensively used in 
image classification. The most common usage of LBPs is to 

1 Matlab code available online: http://tizhoosh.uwaterloo.ca/ 


calculate their histogram and use them as features. Here, for 
sake of comparison, LBPs are extracted and recorded as bar¬ 
codes through concatenation of binary vectors around each 
pixel. As well, the Galoogahi and Sim approach M to apply 
LBP on Radon transformed images, namely local Radon bi¬ 
nary patterns (LRBPs), is also implemented to compare with 
the proposed RBC. Figure [2] shows barcode annotations for 
two medical images from IRMA dataset (see sectio n|5.1| ) for 
different n v values (see lines 4 and 10 in Algorithm mb 

5. EXPERIMENTS 
5.1. Image Test Data 

The Image Retrieval in Medical Applications (IRMA) databas^] 
is a collection of more than 14,000 x-ray images (radio¬ 
graphs) randomly collected from daily routine work at the 
Department of Diagnostic Radiology of the RWTH Aachen 
University |2l] [22]]. All images are classified into 193 cat¬ 
egories (classes) and annotated with the IRMA code which 
relies on class-subclass relations to avoid ambiguities in tex¬ 
tual classification |[22l|23l. The IRMA code consists of four 
mono-hierarchical axes with three to four digits each: the 
technical code T (imaging modality), the directional code D 
(body orientations), the anatomical code A (the body region), 
and the biological code B (the biological system examined). 
The complete IRMA code subsequently exhibits a string of 
13 characters, each in {0,..., 9; a,..., zj: 

2 http://irma-project.org/ 

3 http: //w w w.rad .rwth-aachen. de/ 
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(a) input image 



(b) input image 


-L co de 5 one can establish a suitability measure 77 that prefers 
small mismatch, low error and short codes simultaneously: 

max(nwron g ts ) x max(£? otal ) x max(i* ode ) 

n k = 1 _ ( 4 ) 

' fc.digits F k v Tk 

"•wrong X -C/ total X Jj C oc1c 

Apparently, the larger 77 , the better the method, a desired 
quantification if the code length is important in computation. 
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(C) RBCg 

iiiiiiiiiiiiiiiii 

(e) RBC16 


(g) RBC32 


(i) LBP 


iiiiiiiii 

(d) RBCg 


(f) RBCi 


(h) RBC32 


0) lbp 


(k) LRBP 4 


( 1 ) LRBP 4 


(m) LRBP32 


(n) LRBP32 


Fig. 2. Sample barcodes for RBC, LBP and LRBP 


TTTT-DDD-AAA-BBB. 


( 2 ) 


5.3. Results 


Experiments were performed with all different barcodes. The 
proposed Radon barcode (RBC) with different number of 
projection angles (n p = 4,8,16,32) was trained and tested. 
Among 12,677 IRMA images, 12,631 could be used for train¬ 
ing (some images were ignored due to incomplete codes). The 
training basically annotates each image with a barcode. For 
testing, 1,733 IRMA images were used. For each of the test 
images (that come with their IRMA codes) complete search 
was performed to find the most similar image whereas the 
similarity of an put image J yuery annotated with the corre¬ 
sponding barcode b? uery was calculate based on Hamming 
distance to any other image Ij with its annotated barcode by: 


argmax 

j=l,2,...,1733J#i 


|XOR(bf‘ er »,b j )| 

\ h queryi 


(5) 


For sake of comparison, barcodes generated via local binary 
patterns (LBP) ifTO and local Radon binary pattern (LRBP) 
El were also used to train and test image retrieval. A11 input 
images were resized to 32 x 32 for all barcode approaches. 
LRBP was tested for two projections n p = 4 and n p = 32. 
TableQ]shows ah results. 


More information on the IRMA database and code can be 
found in ETl [2311221 . IRMA dataset offers 12,677 images for 
training and 1,733 images for testing. Figure [3] shows some 
sample images from the dataset long with their IRMA code in 
the format TTTT-DDD-AAA-BBB. 


Table 1. Trained with 12,631 images and tested with 1,733 
images, the number of wrongly retrieved IRMA code digits 
nwrong, the total error E tota \ and the code length L co de are re¬ 
ported. Barcodes are ranked based on their suitability 77 . 


5.2. Accuracy Measurements 

Similar to I22ll24l . the following equation is used to calcu¬ 
late the total error of image retrieval over 1,733 images each 
characterized through 4 IRMA codes with rid digits: 


1733 4 n d 


i=1 k=1 j =1 3 J 


query 


,lj) (3) 


Annotation 

digits 
' t-wrong 

-^total 

-^code 

77 

Rank 

rbc 4 

33.1% 

476.62 

512 

15.526 

1 

RBCg 

33.2% 

478.54 

1024 

7.708 

2 

rbc 16 

32.7% 

470.57 

2048 

3.979 

3 

RBC32 

33.1% 

475.92 

4096 

1.944 

4 

LBP 

32.3% 

463.81 

7200 

1.163 

5 

LRBP4 

33.8% 

483.54 

7200 

1.066 

6 

LRBP32 

34.7% 

501.96 

7200 

1.000 

7 


where rid C {3,4}, and B lk = 10 assuming that every 
digit can have 10 different values 0,1,2,...,9. The function 
S(lj ’ query , /y) delivers 0 if /^’ query = l- otherwise 1. As well, 
the total number of wrong digits in the IRMA codes of re¬ 
trieved images are recorded in nwrong- Using the code length 


Looking at the results of the Table [I] one can state: 

• RBC with 4 Radon projections has the largest 77 (suit¬ 
ability) largely due to its short code length. 
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(f) 1121-200-412-700 (g) 1121-120-918-700 (h) 1121-240-442-700 (i) 1121-220-310-700 (j) 112d-121-500-000 

Fig. 3. Sample images from IRMA Dataset with their IRMA codes TTTT-DDD-AAA-BBB. 


• LRBP has the highest level of error and wrong dig¬ 
its and is ranked lowest according to rj. For LRBP: 

-^total,n p =4 < F/total,n p =8 ^ ^ ^total,n p =32* 

• LBP has the lowest level of error and wrong digits but 
is ranked 5 due to its long (7200 bits) code length. 

• Assuming a total error of 1,733, the difference between 
4 ^ 3 3 1 = 26.76% (for LBP, ranked 5) and 4 ^ 3 g 2 = 
27.50% (for RBC 4 , ranked 1) is rather negligible, hence 
amplifying the role of code length as quantified via rj. 

It should be noted that a comparison with numbers reported in 
literature for using IRMA dates cannot be made for two rea¬ 
sons: 1) literature around IRMA uses as a possible digit 
value for “don’t know” for which 5 = 0.5 is used (Eq. [3]). 
As no classifier is used in this paper due to different nature 
of the barcode-based retrieval, a modified version of the error 
function has been used making a comparison with classifiers 
impossible, and 2 ) a comparison with classifier is not neces¬ 
sary because this paper does not propose to use barcodes as an 
independent approach to CBIR but as a supplementary one. 

5.4. Extra Experiment: ROI Implementation 

The available datasets do not differentiate between inter- and 
intra-class retrieval with respect to feature detection. Assum¬ 
ing that we desire to use barcode annotations to encode re¬ 
gions of interest (ROIs) such as tumors, barcodes could cap¬ 
ture local characteristics of the ROI (Figure]?]). To test ROI- 
based barcodes, 20 breast ultrasound images and their ROIs 
were used (Figure]?] bottom). Only the first hit was examined 
to see whether the barcodes could retrieve benign vs. malig¬ 
nant images correctly (hit or miss). The success rates were 
Yg, and for LRBP, LBP and RBC, respectively. 



Fig. 4. Top: Selected ROI is annotated with a barcode for 
more effective intra-class retrieval. Bottom: Sample benign 
(left) and malignant (right) ROIs. 


6. CONCLUSIONS 

The idea of barcode annotations as auxiliary information be¬ 
side features for medical image retrieval was proposed in this 
paper. Radon projections are thresholded to assemble bar¬ 
codes. IRMA x-ray dataset with 14,410 images for training 
and testing was used to verify the performance of barcode- 
based image retrieval. Radon barcodes appear to provide 
short but expressive codes useful for medical image retrieval. 
Future work should investigate ROI barcodes for intra-class 
retrieval whereas inter-class retrieval is performed by other 
methods such as bag-of-feature classification paradigms. 
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